European COVID-19 Forecast Hub Evaluation Report for Estonia
2021-05-31
Forecast visualisation (Estonia)
Forecast visualisations. The date of the tab marks the date on which a forecast was made (only last 4 weeks shown).
2021-05-24
Cases
Deaths
2021-05-17
Cases
Deaths
2021-05-10
Cases
Deaths
2021-05-03
Cases
Deaths
2021-04-26
Cases
Deaths
Forecast scores (Estonia)
Scores separated by target and forecast horizon. Only models with submission in the last 4 are shown.
Cases
1 week ahead horizon
2 weeks ahead horizon
3 weeks ahead horizon
4 weeks ahead horizon
Deaths
1 week ahead horizon
2 weeks ahead horizon
3 weeks ahead horizon
4 weeks ahead horizon
Evaluation metrics
- The first column (n) gives the number of forecasts included in the evaluation
- Relative skill (rel_skill) is a metric based on the weighted interval score (WIS) that is using a ‘pairwise comparison tournament’. All pairs of forecasters are compared against each other in terms of the weighted interval score. The mean score of both models based on the set of common targets for which both models have made a prediction are calculated to obtain mean score ratios. The relative skill is the geometric mean of these mean score ratios. Smaller values are better and a value smaller than one means that the model beats the average forecasting model.
- Coverage (50% Cov. / 95% Cov.) is the proportion of observations that fell within a given prediction interval. Ideally, a forecast model would achieve 50% coverage of 0.50 (i.e., 50% of observations fall within the 50% prediction interval) and 95% coverage of 0.95 (i.e., 95% of observations fall within the 95% prediction interval). Values of coverage greater than these nominal values indicate that the forecasts are underconfident, i.e. prediction intervals tend to be too wide, wherease values of coverage smaller than these nominal values indicate that the forecasts are overconfident, i.e. prediction intervals tend to be too narrow.
- The weighted interval score (wis) is a proper scoring rule (i.e., it cannot be “cheated”) that is suited to scoring forecasts in an interval format. It has three components: sharpness, underprediction and overprediction. Sharpness is the width of the prediction interval. Over- and underprediction (overpred/underpred) only come into play if the prediction interval does not cover the true value. They are the absolute value of the difference between the upper or lower bound of the prediction interval (depending on whether the forecast is too high or too low).
- bias is a measure between -1 and 1 that expresses the tendency to underpredict (-1) or overpredict (1). In contrast to the over- and underprediction components of the WIS it is bound between -1 and 1 and cannot go to infinity. It is therefore less susceptible to outliers.
- aem is the absolute error of the median forecasts. A high aem means the median forecasts tend to be far away from the true values.
Scores over time (Estonia)
Visualisation of the weighted interval score over time. In addition, the components of the interval score, sharpness (how narrow are forecasts - smaller is better), and penalties for underprediction and overprediction are shown. Scores are again separated by forecast horizon
1 week ahead horizon
Weighted interval score
Overprediction
Underprediction
Sharpness
2 weeks ahead horizon
Weighted interval score
Overprediction
Underprediction
Sharpness
3 weeks ahead horizon
Weighted interval score
Overprediction
Underprediction
Sharpness
4 weeks ahead horizon
Weighted interval score
Overprediction
Underprediction
Sharpness
If you want to learn more about a model, you can go the the ‘data-processed’-folder of the European Forecast Hub github repository, select a model and access the metadata file with further information provided by the model authors.